Image capture in a vibrational environment

ABSTRACT

This disclosure describes optimizing a clarity of images captured by a camera exposed to vibrations, such as a camera mounted on an aerial vehicle. The vibrations can be caused by rotors, motors, forces (e.g., lift, drag, etc.) acting on the UAV, environmental factors (e.g., wind, turbulence, etc.), or any other force that may cause asymmetry. An inertial measurement unit can measure the vibrations and determine a vibrational pattern imposed upon the camera. The inertial measurement unit can identify one or more dead points in the vibrational pattern, and times associated therewith. The inertial measurement unit can send the one or more dead points and/or the times associated therewith to the camera, and cause the camera to capture and/or store images at times corresponding to the one or more dead points to enable capture of images with little or no blur.

RELATED APPLICATIONS

This application claims priority to and is a continuation of U.S. patentapplication Ser. No. 15/197,550, filed on Jun. 29, 2016, the entirecontents of which are incorporated herein by reference.

BACKGROUND

Unmanned aerial vehicles (UAVs) have become increasingly popular. UAVsof all shapes and sizes are constructed for a myriad function. Among themany popular functions of UAVs is aerial photography. However, takingpictures from a UAV can pose many problems, such as blur due tovibrations generated during operation of the UAV.

Traditionally, aerial photography systems use vibration dampeninghardware in an attempt to lessen the blur caused by vibrations. However,the vibration dampening hardware can be heavy and cumbersome, therebyaffecting flight characteristics of the UAV, such as decreasing payloadavailable, increasing drag and decreasing range. These types ofdampening hardware may be especially ill suited for use on small UAVs ormicro UAVs that have little or no capacity for payload.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Thesame reference numbers in different figures indicate similar oridentical items.

FIG. 1 is a pictorial flow diagram of an example process of an imagecapture optimization system.

FIG. 2 is a schematic diagram showing an example environment where animage capture optimization system may anticipate vibrational changes tobe imposed on a camera due to changing flight conditions.

FIG. 3 is a graphical diagram of an example vibrational pattern of acamera and images captured at various points on the vibrational graph.

FIG. 4 is a pictorial flow diagram of an example process of an imagecapture optimization system in which high frequency vibrations areimposed on the camera.

FIG. 5 is a flow diagram showing an illustrative process to capture animage during one or more dead points of a vibrational pattern.

FIG. 6 is a flow diagram showing an illustrative process to evaluatecaptured images based on timing associated with one or more dead pointsof a vibrational pattern.

FIG. 7 is a flow diagram showing an illustrative process to captureimages during periods of high frequency vibrations.

FIG. 8 illustrates example components of a UAV configured with an imagecapturing optimization system.

DETAILED DESCRIPTION

This disclosure provides methods, apparatuses, and systems foroptimizing clarity of images captured by a camera exposed to vibrations.In some embodiments, the camera can be mounted on an aerial vehicle,such as an unmanned aerial vehicle (UAV). In such embodiments, thevibrations can be caused by rotors, motors, forces (e.g., lift, drag,etc.) acting on the UAV, environmental factors (e.g., wind, turbulence,etc.), and/or any other force that may cause asymmetry. For example, aquad-copter configured for aerial photography (e.g., with a camerasystem) may include four rotors, each coupled to an airframe by a spar.As each of the rotors rotate to generate lift, the quad-copter mayexperience a vibration caused by the rotors passing the respectivespars. The vibration may be transmitted to the camera system mounted onthe airframe.

The UAV can include an inertial measurement unit (IMU) to measure avibrational pattern experienced by the airframe and/or the camera. Thevibrational pattern can be harmonic, e.g., constant over time. Forexample, a UAV flying straight and level may experience substantiallythe same harmonic vibrations throughout the flight. For another example,a UAV executing maneuvers in flight (e.g., turns, climbs, descents,etc.) may experience different vibrations over time. As such, thevibrational pattern over time may be random.

The IMU can measure accelerations and dead points in the vibrations. Thedead points can be short periods of time (e.g., 1/60 second, 1/240second, etc.) in which there is minimal to no acceleration in thevibrational pattern. In various examples, the IMU can send a signal tothe camera to capture images (e.g., entire images, pixels of an image,etc.) during one or more of the dead points in the vibrations. In someexamples, the IMU can send the times associated with dead points to acomputing system configured to post-process a plurality of images. Insuch examples, the computing system may receive the plurality of imagesfrom the camera, store the images captured during dead points, anddiscard other images.

In various examples, the UAV and/or the camera may experience highfrequency vibrations. In such examples, the dead points may span ashorter time than during low frequency vibrations. In some examples, thecamera may increase a shutter speed in order to capture a plurality ofimages corresponding to dead points. Due to the increased shutter speed,the plurality of images captured may be underexposed. In some examples,the IMU may send times associated with the dead points to the camera,and the camera may pre-process the underexposed images by comparing thetimes associated with the dead points to a capture time of each of theimages. Responsive to determining that a capture time corresponds to adead point, the camera can store an image. Additionally, the camera candiscard images that do not correspond to dead points. In such examples,the camera may send the images to the computing system forpost-processing, such as blending (e.g., combining) the multipleunderexposed images.

In various examples, the camera may send the plurality of underexposedimages to the computing system for pre and post-processing. In suchexamples, the IMU may send times associated with the dead points to thecomputing system. The computing system may compare the times associatedwith the images to the times associated with the dead points, anddiscard images that do not correspond to dead points. The computingsystem may blend the remaining underexposed images into a darker orricher (e.g., fully exposed) image.

In various examples, the camera may receive a signal from an imagesensor. The signal can include a period of time in which the camerashutter, such as an electronic shutter, is to remain open to collectlight. The camera can receive times associated with the dead points fromthe IMU. In various examples, the times associated with the dead pointscan include a sequence of times, such as, for example, a dead pointoccurs every 5 milliseconds (ms), every 10 ms, etc. Responsive toreceiving the times, the camera can collect pixels at times of minimalor no vibration. The pixels collected over the shutter cycle cancomprise a collected image. In various examples, the camera may send thecollected image to the computing system for post-processing. In suchexamples, the post processing can include a determination that thecollected image is underexposed. Based on the determination that thecollected image is underexposed, the computing system can cause theimage sensor to elongate the shutter cycle (e.g., instruct the shutterto remain open longer).

The apparatuses, techniques, and systems described herein may beimplemented in a number of ways. Though described with reference toUAVs, the image capturing optimization system is not limited to use withUAVs. Instead, the image capturing optimization system may beimplemented on many other types of flying (e.g., manned aircraft,spacecraft, etc.) or non-flying vehicles (e.g., autonomous cars, trucks,tractors, bicycles), hand-held cameras, and the like. Exampleimplementations of the image capturing optimization system are providedbelow with reference to the following figures.

FIG. 1 is a pictorial flow diagram of an example process 100 of an imagecapture optimization system. At 102, a UAV 108 can identify avibrational pattern 110 imposed upon an airframe of the UAV 108. Thevibrational pattern 110 can be caused by rotors, motors, forces actingon the UAV 108 (e.g., lift, drag, etc.), environmental factors (e.g.,wind, turbulence, etc.), and/or any other force that may causeasymmetry. The vibrational pattern can be identified by an inertialmeasurement unit (IMU) in the UAV 108. In various examples, the IMU canbe mounted in the airframe of the UAV 108. In some examples, the IMU canbe mounted in, on, and/or proximate to a camera 112 used to collectimages 114.

In various examples, the surface on which the camera 112 is mounted(e.g., an airframe, a fuselage, a wing, etc.) may experience asubstantially uniform vibrational pattern. In such examples, the UAV 108may include an IMU for each surface with a substantially uniformvibrational pattern. For example, a quad-copter airframe on which one ormore cameras may be mounted may experience a substantially uniformvibrational pattern. As such, the quad-copter may only require one IMUto measure the vibrational pattern and to transmit vibrational patterninformation to the camera(s) and/or the computing system. For anotherexample, a fixed wing aircraft may experience a first vibrationalpattern on a first wing and a second vibrational pattern on a secondwing. In such an example, the fixed wing aircraft may include a firstIMU for one or more cameras mounted on the first wing and a second IMUfor one or more cameras mounted on the second wing.

At 104, the IMU can determine dead points 116 of the vibrational pattern110. The dead points 116 can include time windows of the vibrationalpattern 110 in which there is little to no acceleration. The time windowcan represent a period of time in which a camera can capture a clear andcrisp image (e.g., an image with little to no blur caused by camerashake). In FIG. 1, the time windows are illustrated as t₁, t₂, and t₃.In other examples, the time windows can be longer or shorter. Forexample, the time windows of a high frequency vibration can besignificantly shorter. For another example, the time windows of anultra-low frequency vibration can be longer.

At 106, the camera on the UAV 108 can capture images 114. To capture animage, the camera 112 triggers a shutter to open and close (e.g.,shutter cycle). In various examples, the shutter can be a mechanicalshutter. In such examples, the mechanical shutter can use shuttercurtains which open and close to produce an exposure. In some examples,the shutter can be an electronic shutter. In such examples, the shuttercycle can occur by engaging an image sensor (e.g., turning the imagesensor on and off) to control exposure and/or a rate of pixel capture.The image sensor can be a charge coupled device (CCD), a complementarymetal oxide semiconductor (CMOS), or other image sensor. In variousexamples, the IMU can send the camera 112 a signal including the timewindows (e.g., times associated with the dead points). In such examples,responsive to the signal, the camera 112 can trigger the shutter tocycle during the time windows, thereby capturing clear and crisp imagesand/or pixels of images that include little or no blur.

In various examples, the camera 112 can first capture a plurality ofimages, and then receive the signal including the time windows from theIMU. In such examples, the camera 112 can store images captured duringthe time windows, and discard the other images. In some examples, theIMU can send the signal including the time window and the camera 112 cansend the plurality of images to the computing system. In such examples,the computing system can compare times associated with each of theplurality of images to the time windows. The computing system can thenstore images captured during the time windows, and discard other images.

FIG. 2 is a schematic diagram showing an example environment 200 wherean image capture optimization system may anticipate vibrational changesto be imposed on a camera. The UAV 202 has an airframe 204, one or morerotors 206, and one or more motors 208. In some examples, the UAV 202may include a fuselage (e.g., airframe) and two or more fixed wingsprotruding from the airframe. In such examples, the UAV 202 may alsoinclude vertical and/or horizontal stabilizers coupled to the airframe.

The airframe 204 may comprise carbon fiber, titanium, aluminum, plastic,combinations thereof, or any other material appropriate for aircraftconstruction. In various examples, the airframe 204 may comprise rotorguards to protect the one or more rotors 206. In such examples, therotor guards may cause some of the vibrations experienced by theairframe. In some examples, rotor guards can be substantially circular,and in the plane of the rotors 206. In some examples, the rotor guardsmay comprise a structure which may envelop part of or all the UAV 202.For example, the rotor guard may include a truss structure that issubstantially circular, surrounds the UAV, and is able to absorb impactforces.

In the illustrative example, the UAV 202 is a quad-copter with fourrotors 206 and four motors 208. In some examples, the UAV 202 maycomprise a lesser or greater number of rotors 206, such as, for example,a tri-copter or an octo-copter. In some examples, the UAV 202 maycomprise a lesser or greater number of motors 208 that are coupled toand configured to drive rotors 206. For example, the UAV 202 maycomprise a single motor coupled to and configured to drive the one ormore rotors 206. In various examples, motors 208 may comprise electricalmotors. In such examples, the electric motors may be powered by anyreasonable source of electrical power, such as, for example, lithium-ionbatteries, fuel cells, solar power, nuclear power, or a hybridtechnology. In some examples, the motors 208 may comprise combustionengines, in some examples.

In various examples, the motors 208 and/or the rotors 206 may be coupledto the airframe 204 via a spar 212. The spar 212 may comprise carbonfiber, titanium, aluminum, plastic, combinations thereof, or any othermaterial appropriate for aircraft construction. In some examples, therotor 206 passing over the spar 212 may cause some of the vibrationsexperienced by the airframe.

The motors 208 may produce power which is transmitted to the rotors 206via a drive shaft in order to produce thrust for propulsion. The driveshaft may comprise a metal material (e.g., aluminum, steel, stainlesssteel, titanium, alloys thereof, etc.), a plastic material (e.g.,high-density polyethylene, acrylic, melamine, polycarbonate, etc.), acomposite material (e.g., fiberglass, carbon fiber, etc.), a woodmaterial, and combinations of the foregoing, among others. In someexamples, the drive shaft may cause some of the vibrations experiencedby the airframe.

In the illustrative example, the rotors 206 are the same size and/orshape (e.g. chord, thickness, and/or wingspan). In some examples, therotors 206 may be different sizes and/or shapes. For example, the rotors206 on a fore-end of the airframe may have a larger wingspan than therotors 206 on an aft-end of the airframe, or vice versa.

The rotors 206 may comprise a composite material, a wood material, aplastic material, a metallic material, or a combination thereof. Invarious examples, the rotors 206 may be variable speed, variable pitchrotors. In other examples, the rotors 206 may be variable speed, fixedpitch rotors. In yet other embodiments, the rotors 206 may be fixedspeed, variable pitch rotors. Additionally or alternatively, variousexamples may include one or more of the foregoing rotors used incombination with one or more of a different foregoing rotor, or otherpropulsion systems.

In various examples, the speed and/or pitch of the rotors may bedetermined by a computing system 214 based upon input from one or moreinertial sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.)and/or positioning sensors (e.g., global positioning sensors, ultrasonicsensors, radar systems, etc.). The inertial sensors may be configured tomeasure precise positioning data of the UAV 202 along three axes:heading, roll and pitch, and send the positioning data to the computingsystem 214.

In various examples, the computing system 214 can receive flightmanagement input (e.g., input to control direction, velocity, altitude,waypoints, geolocation, etc.) from a remote operator. For example, theremove operator may input changes to a flight path 218 (e.g., a path oftravel of a vehicle). In some examples, the UAV 202 can be configured tofly autonomously. In such examples, the flight management input caninclude a flight plan stored locally on the UAV 202 and accessed by thecomputing system 214. The flight plan may comprise a series of waypointsdetermined by a latitude, longitude, and altitude, a position on a3-dimensional grid system, or a combination thereof. The series ofwaypoints can determine the flight path 218 of the UAV 202. In someexamples, the flight path 218 can be based on images desired to becaptured by a camera 216. For example, one or more of the series ofwaypoints may correspond to desired views of an image and/or objects tobe captured in various images.

In various examples, the computing system 214 can evaluate the flightmanagement input (e.g., from the remote operator and/or the flightplan), and anticipate changes in vibrations based on changes in aircraftattitude, rotor speed, rotor pitch, motor speed, and the like. Asillustrated in FIG. 2, flight path 218 may include a first section218(1) corresponding to straight and level flight, and a second section218(2) corresponding to a level right-hand turn. During the firstsection 218(1), the airframe 204 may experience a first vibrationalpattern 220(1). The first vibrational pattern 220(1) may be measured byan IMU 222 of the UAV 202.

The vibrational pattern 220 can represent a movement m (shown aspositive and negative values from a neutral position) of an object(e.g., the camera, the IMU, etc.) over a time t. An amount of movement mcan be represented by an amplitude 230 of the vibrational pattern 220.The vibrational pattern 220 can include a combination of vibrations frommultiple sources (e.g., rotors, spars, motors, rotor guards, aerodynamicforces etc.). In various examples, the vibrational pattern 220 caninclude vibrations from the multiple sources in one or more axes 224perpendicular to the camera. For example, the camera may be orientedalong axis Z, and the vibrational pattern may include vibrationsoccurring about axis X and/or axis Y. Additionally or alternatively, thevibrational pattern can include vibrations from the multiple sourcesalong the camera orientation axis Z.

While traveling along the first section 218(1), the IMU 222 can measureone or more dead points 226(1) and corresponding time windows. Invarious examples, the IMU 222 can send a signal including the deadpoints 226(1) and/or corresponding time windows to the camera 216. Insome examples, the signal can be sent via the computing system 214.Based on the signal, the camera can capture images during the deadpoints 226(1). In some examples, the IMU 222 can send the dead points226(1) and/or corresponding time windows to the computing system 214. Insuch examples, the computing system 214 can process images from thecamera 216 and discard images captured outside of the time windowsassociated with the dead points 226(1).

In various examples, the computing system 214 can evaluate the flightmanagement input and determine that the UAV 202 is at (e.g., directinput from a remote operator) and/or approaching (e.g., based on theflight plan) a waypoint 228 (e.g., a point at which an attitude,direction, and/or altitude of the UAV changes) between the first section218(1) and the second section 218(2) of the flight path 218. In suchexamples, the computing system 214 can determine the adjustments to themotors 208, rotors 206, camera 216, attitude of the UAV 202, and/or anyother factors to effect a maneuver at the waypoint 228, and cananticipate the second vibrational pattern 220(2).

In various examples, the second vibrational pattern 220(2) can include avibrational pattern 220 of higher, lower or equal amplitude 230 and/orfrequency, as compared to the first vibrational pattern 220(1). In theillustrative example, the second vibrational pattern 220(2) includes asubstantially similar amplitude 230 to the first vibrational pattern220(1), and a higher frequency. Based at least in part on the higherfrequency, the dead points 226(2) and corresponding time windows aresmaller (e.g., a shorter time window) than the dead points 226(1) of thefirst vibrational pattern 220(1).

In some examples, based on the anticipated second vibrational pattern220(2), the computing system can send a signal including the anticipateddead points 226(2) and/or corresponding time windows to the camera 216.Based on the signal, the camera can capture images during theanticipated dead points 226(2). In such examples, the anticipated deadpoints 226(2) can be used to determine when to capture images until theIMU 222 measures the actual vibrational pattern 220 and correspondingdead points 226 associated with the second section 218(2) of the flightpath 218.

Though FIG. 2 is described with respect to a UAV, as stated above, theuse of the camera system as described is not limited to such anapplication. For example, a camera, an IMU, and a computing system maybe included in an autonomous vehicle. The computing system cananticipate a change in the path of travel of a vehicle (e.g., flightpath), such as an upcoming turn, and can anticipate a vibrationalpattern associated therewith. The computing system can use theanticipated vibrational pattern to calculate anticipated dead points,and can trigger a camera to capture images and/or pixels of images atthe anticipated dead points.

FIG. 3 is a graphical diagram of an example vibrational pattern 300 of acamera and images 302 captured at various points on the vibrationalgraph. In various examples, the vibrational pattern 300 can includevibrations experienced by the camera in the two axes perpendicular tothe axis in which the camera (e.g., lens) is directed. In some examples,the vibrational pattern 300 can include vibrations experienced by thecamera about three axes (e.g., X, Y, Z). In various examples, thevibrational pattern 300 can be caused by rotors, motors, forces (e.g.,lift, drag, etc.) acting on a UAV to which the camera is coupled,environmental factors (e.g., wind, turbulence, etc.), and/or any otherforce that may cause asymmetry. The vibrational pattern 300 can beidentified by an inertial measurement unit (IMU). In various examples,the IMU can be mounted in an airframe of the UAV. In some examples, theIMU can be mounted in, on, and/or proximate to the camera.

In various examples, the IMU can determine dead points 304, such as deadpoints 226, of the vibrational pattern 300. The dead points 304 caninclude time windows of the vibrational pattern in which there isminimal (e.g., little to none) acceleration. The time window canrepresent a period of time in which a camera can capture a clear image(e.g., an image with little to no blur due to camera shake). In someexamples, the IMU can send a signal with the dead points 304 andcorresponding time windows to the camera. In such examples, the cameracan selectively capture images 302(1), 302(5) during the time window(e.g., capture clear images).

In various examples, the camera can capture a plurality of images 302throughout a timespan t. In such examples, the camera can process theplurality of images 302 to determine whether a time associated with eachof the plurality of images 302 corresponds to a dead point 304.Responsive to a determination that images 302(1) and 302(5) werecaptured during a dead point 304, the camera can store the images 302(1)and 302(5), and discard the other images 302(2), 302(3), 302(4), 302(6)and 302(N). In some examples, the camera can send the images 302(1) and302(5) to a computing system and/or a data store on the UAV.

In some examples, such as in a camera with an electronic shutter, thecamera can open the electronic shutter for a designated period (e.g.responsive to a signal from the CCD, CMOS, and/or other image sensor) tocapture light in order to collect pixels of an image 302. In suchexamples, the camera can receive the signal with the dead points 304from the IMU, and collect pixels corresponding to the dead points 304.For example, the shutter can be open for a period of 100 ms. The IMU cansend a signal to the camera indicating a dead point 304 every 5 ms.Responsive to the signal from the IMU, the camera can selectivelycollect the pixels at the dead points 304.

In various examples, the image processing can be completed by acomputing system. In such examples, the computing system can receive theplurality of images 302 from the camera, and the dead points 304 fromthe IMU. The computing system can determine that the images 302(1) and302(5) and/or pixels of an image correspond to dead points 304, and canstore the images 302(1) and 302(5) and/or pixels of the image. In someexamples, the computing system can send the images 302(1) and 302(5) toan external data store via a communication channel, such as via a Wi-Fior Bluetooth® signal.

FIG. 4 is a pictorial flow diagram of an example process 400 of an imagecapture optimization system in which high frequency vibrations areimposed on a camera.

At 402, a UAV can collect data using an IMU. The IMU can measure avibrational pattern 408 imposed upon the camera. In various examples,the IMU can be mounted in an airframe of a UAV 408. In some examples,the IMU can be mounted in, on, and/or proximate to the camera. Thevibrational pattern 410 can be a pattern of any amplitude and/orfrequency. The vibrational pattern 410 can be harmonic (e.g., constantover time) or random (e.g., changes over time). In the illustrativeexample, the vibrational pattern 410 represents a harmonic, highfrequency vibration. The vibrational pattern 410 can be caused byrotors, motors, forces (e.g., lift, drag, etc.) acting on the UAV,environmental factors (e.g., wind, turbulence, etc.), or any other forcethat may cause asymmetry.

In various examples, the IMU can send a signal with the vibrationalpattern 410 to the camera. In such examples, the camera can determinedead points in the vibrational pattern 410. In some examples, the IMUcan determine the dead points in the vibrational pattern 410, andinclude the dead points in the signal. The dead points can include timewindows of the vibrational pattern 410 in which there is minimalacceleration. The time window can represent a period of time in which acamera can capture a clear image. The time windows associated with ahigh frequency vibration can be short periods of time. As such, a speedof the camera shutter cycle (e.g., open and close) must be fast to fitwithin the time window. Due to the speed of the fast shutter cycle, thecamera may capture underexposed images (e.g., images captured withlimited amount of light).

At 404, the UAV can collect imagery using a camera. The imagery caninclude an image and/or pixels corresponding to an image. In variousexamples, such as in a high frequency vibrational environment, thecamera can capture a plurality of images 412 and/or pixels of images. Insome examples, based in part on a fast capture used in the highvibrational environment, the plurality of images 412 and/or pixels ofimages can comprise underexposed images. The camera can capture theimages and/or pixels of images at a fixed rate (e.g., 20 frames persecond, 6 frames per second, 240 images per second, 480 images persecond, etc.), or at a random rate.

In some examples, such as with an electronic shutter, the camera cancycle the shutter and/or set a pixel capture rate based on a signal froma CCD, a CMOS, or other image sensor. In such examples, the electronicshutter can remain open for a period of time, as determined by the imagesensor. The camera can receive the signal from the IMU comprising thevibrational pattern 410 and/or the dead points and corresponding timewindows. The camera can evaluate the time the shutter is open, andcollect pixels corresponding to the time windows associated with deadpoints. As such, the camera can collect pixels of an image at intervalscorresponding to the dead points (e.g. a time sequence corresponding tothe time windows associated with the dead points). Based on the rate ofpixel collection (e.g., every 5 ms during an open period of 100 ms), thepixels of the image can be underexposed images, as only a portion of thelight is captured on the pixels at the rate of pixel collection.

In various examples, such as in examples with a mechanical shutter, thecamera can capture the images by cycling the mechanical shutter. In suchexamples, the camera can receive the signal from the IMU comprising thevibrational pattern 410 and/or the dead points and corresponding timewindows. The camera can compare the times associated with theunderexposed images to the times associated with the dead points. Thecamera can store the images 412 corresponding to dead points, anddiscard images that do not correspond to dead points. In variousexamples, the camera can send the stored images 412 to a computingsystem of the UAV 408.

Alternatively, the computing system can perform the pre-processing ofthe images. In various examples, the computing system can receive thesignal comprising the vibrational pattern 410 and/or the dead points andcorresponding time windows from the IMU. Additionally, the computingsystem can receive the plurality of images 412 from the camera, e.g.,the images corresponding to dead points and the images not correspondingto dead points. The computing system can compare the times associatedwith the images 412 to the time windows associated with the dead points.The computing system can store the underexposed images corresponding todead points 414, and discard images that do not correspond to deadpoints.

At 406, the computing system can process the imagery using a computingsystem. The computing system can determine that the images and/or pixelsof images are underexposed. In various examples, such as with anelectronic shutter, the computing system can determine that thecollected pixels are not fully exposed (e.g., pixels are not at therequisite exposure for a computing algorithm). In such examples, thecomputing system can send a signal to the image sensor to instruct thecamera to open the shutter for a longer period. Responsive to the longershutter open period, the camera can capture more light on the pixels,thereby increasing the exposure.

In some examples, the computing system can process the imagescorresponding to dead points 414 by blending the images 412 (e.g.,underexposed images) into a fully exposed (e.g., darker) image 416. Invarious examples, the computing system can store the fully exposed image416 in a local data store. In some examples, the computing system cansend the fully exposed image 416 to an external computing device, suchas via a communications channel. In other examples, the computing systemon the UAV 408 can send the underexposed images 414 to an externalcomputing system for processing into a fully exposed image 416.

FIGS. 5-7 are flow diagrams of illustrative processes. The processes areillustrated as a collection of blocks in a logical flow graph, whichrepresent a sequence of operations that can be implemented in hardware,software, or a combination thereof In the context of software, theblocks represent computer-executable instructions stored on one or morecomputer-readable storage media that, when executed by one or moreprocessors, perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, data structures, and the like that perform particularfunctions or implement particular abstract data types. The order inwhich the operations are described is not intended to be construed as alimitation, and any number of the described blocks can be combined inany order and/or in parallel to implement the processes. The processesdiscussed below may be combined in any way to create derivativeprocesses that are still within the scope of this disclosure.

FIG. 5 is a flow diagram showing an illustrative process 500 to capturean image during one or more dead points of a vibrational pattern.

At block 502, an IMU identifies a vibrational pattern of a UAV. Thevibrational pattern can include a combination of vibrations frommultiple sources (e.g., rotors, spars, motors, rotor guards, aerodynamicforces etc.). In various examples, the vibrational pattern can includevibrations from the multiple sources in one or more axes perpendicularto the camera. For example, the camera may be oriented along axis Z, andthe vibrational pattern may include vibrations occurring about axis Xand/or axis Y. Additionally or alternatively, the vibrational patterncan include vibrations from the multiple sources along the cameraorientation axis Z.

The vibrational pattern can be substantially harmonic, e.g., constantover time. For example, a UAV flying straight and level may experiencesubstantially the same vibrations throughout the flight. As such, thevibrational pattern may be harmonic. For another example, a UAVexecuting maneuvers in flight (e.g., turns, climbs, descents, etc.) mayexperience different vibrations over time. As such, the vibrationalpattern over time may be random (e.g., changes over time). Thevibrational pattern can include accelerations and dead points (e.g.,areas of minimal or no acceleration).

In various examples, a computing system of the UAV can evaluate flightmanagement input (e.g., from a remote operator and/or a flight plan),and anticipate changes in vibrations based on changes in aircraftattitude, rotor speed, rotor pitch, motor speed, and the like. In suchexamples, the computing system can identify an anticipated vibrationalpattern of a UAV.

At block 504, the IMU or the computing system can determine one or moredead points of the vibrational pattern. In various examples, the one ormore dead points can be detected by an accelerometer. In such examples,the accelerometer can measure accelerations in vibrations to includechanges in accelerations in a positive and a negative direction. The oneor more dead points can be the points between the positive and negativeaccelerations.

In some examples, the computing system can receive the accelerationsfrom the accelerometer, and can identify the one or more dead points. Insuch examples, the computing system can identify the accelerations inthe positive and negative directions, and the points between thepositive and negative accelerations (e.g., dead points). The one or moredead points can include respective time windows of the vibrationalpattern in which there is little to no acceleration (e.g., pointsbetween positive and negative accelerations). The time window canrepresent a period of time in which a camera can capture a clear image(e.g., an image with little to no blur due to camera shake).

At block 506, the IMU or the computing system can send a signal to thecamera comprising the one or more dead points. The signal can includethe time windows corresponding to the dead points.

At block 508, responsive to the signal, the camera can capture an imagebased on the one or more dead points and/or the corresponding timewindow. The camera can cycle a shutter (e.g., open and close) to capturethe image during the time window.

FIG. 6 is a flow diagram showing an illustrative process 600 to evaluatecaptured images based on timing associated with one or more dead pointsof a vibrational pattern.

At block 602, a camera on a UAV can capture an image. The camera cancapture the image by cycling a shutter at a time. In various examples,the camera can save the image with the time associated with the shuttercycle. In some examples, the camera can capture images at a periodicrate. For example, the camera can capture 60 images per minute. Foranother example, the camera can capture 180 images per minute. In otherexamples, the camera can capture images at a non-periodic rate.

At block 604, the camera can determine that the time associated with theimage capture corresponds to one or more dead points. The one or moredead points can include respective time windows of the vibrationalpattern in which there is little to no acceleration. A time window canrepresent a period of time in which a camera can capture a clear image(e.g., an image with little to no blur due to camera shake). In variousexamples, the camera can receive a signal with the one or more deadpoints and/or time windows associated therewith from an IMU.

At block 606, the camera can store the image. In various examples, thecamera can store the image in a memory of the camera. In some examples,the camera can send the image to the computing system for storage in adatastore.

FIG. 7 is a flow diagram showing an illustrative process 700 to captureunderexposed images during periods of high frequency vibrations.

At block 702, a camera management system can determine that avibrational pattern imposed on a camera is a high frequency vibration.In various examples, an IMU of the camera management system can measurethe high frequency vibration.

At block 704, the camera management system can cause the camera tocapture a plurality of images. In various examples, the cameramanagement system can cause the camera to capture the plurality ofimages at a high shutter speed. Due to the high shutter speed, theplurality of images captured may be underexposed. In some examples, theIMU may send times associated with the dead points to the camera, andthe camera may pre-process the plurality of images, discarding theimages that do not correspond to dead points. In such examples, thecamera may send the images to the computing system for post-processing,such as blending (e.g., combining) the multiple underexposed images.

In various examples, the camera management system, such as via a CCD,CMOS, or other image sensor, can cause a camera shutter to remain openfor a given period of time to capture pixels (e.g., accumulate an amountof light on the pixels). In such examples, the camera can compare timesassociated with the dead points to the pixels, and collect the pixelsthat correspond to the times associated with the dead points.

At block 706, the camera management system can determine that a firstgroup of the plurality of images and/or pixels corresponds to aplurality of dead points. In various examples, the IMU can determine avibrational pattern associated with the high frequency vibrations. Insuch examples, the IMU can identify a plurality of dead points on thevibrational pattern. Additionally, the IMU can identify a time windowassociated with each of the respective dead points.

In various examples, the IMU can send a signal with the dead points andcorresponding time windows to the camera. In such examples, the cameracan compare a time associated with the capture of each of the imagesand/or pixels to the one or more dead points. The camera can storeimages and/or pixels with capture times that correspond to therespective time windows, and can discard other images and/or pixels. Insome examples, the camera can send the plurality of images and/or pixelsto a computing system for post-processing. In such examples, thecomputing system can receive the dead points and corresponding timewindows from the IMU, and can compare the times associated with theplurality of images and/or pixels to the dead points. The computingsystem can store images and/or pixels with capture times associated withdead points, and can discard the other images.

At block 708, the camera management system can process the first groupof the plurality of images and/or pixels. In some examples, such as inexamples with mechanical shutters, the processing can include blendingthe first group of the plurality of images into a single image.

In other examples, such as in examples with electronic shutters, theprocessing can include evaluating an exposure of the pixels of an imageto determine whether the image is sufficiently exposed. Responsive to adetermination that the image is sufficiently exposed, the cameramanagement system can cause the image to be saved to a data store.Responsive to a determination that the image is underexposed, the cameramanagement system can cause the camera, such as through the imagesensor, to increase a time in which the shutter remains open during ashutter cycle. Due to an increased number of instances of capturinglight during the elongated shutter cycle, the pixels of the image maycapture more light, thereby increasing an exposure of the image.

FIG. 8 illustrates example components of a UAV 800 configured with animage capturing optimization system. The UAV 800 may be any type ofunmanned aerial vehicle, such as a fixed or rotary wing vehicle.

In the illustrated example, the UAV 800 includes at least one processor802, at least one memory 804, a camera 806, an inertial measurement unit808, and one or more communication interface(s) 810. Each processor 802may itself comprise one or more processors or processing cores. Forexample, the processor 802 can be implemented as one or moremicroprocessors, microcomputers, microcontrollers, digital signalprocessors, central processing units, state machines, logic circuitries,and/or any devices that manipulate signals based on operationalinstructions. In some cases, the processor 802 may be one or morehardware processors and/or logic circuits of any suitable typespecifically programmed or configured to execute the algorithms andprocesses described herein. The processor 802 can be configured to fetchand execute computer-readable processor-executable instructions storedin the memory 804. The processor(s) 802 may be operably coupled to thememory 804 via bus 312, which in some instances can include one or moreof a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus,and any variety of local, peripheral and/or independent buses.

The memory 804 may include a tangible non-transitory computer storagemedia and may include volatile and nonvolatile memory and/or removableand non-removable media implemented in any type of technology forstorage of information such as computer-readable processor-executableinstructions, data structures, program modules or other data. The memory804 may include, but is not limited to, RAM, ROM, EEPROM, flash memory,solid-state storage, magnetic disk storage, optical storage, and/orother computer-readable media technology.

The memory 804 may be used to store and maintain any number offunctional components that are executable by the processor 802. In someimplementations, these functional components comprise instructions orprograms that are executable by the processor 802 and that, whenexecuted, implement operational logic for performing the actions andservices attributed above to the UAV 800. Functional components of theUAV 800 stored in the memory 804 may include an operating system 814 acomputing system 816, and a datastore 818. The operating system 814 ofthe UAV 800 may be any operating system capable of managing computerhardware and software resources.

In various examples, the computing system 816 may include a controlmanagement system configured to adjust one or more control inputs (e.g.,rotor speed, rotor pitch, motor speed, flight control inputs, etc.)necessary to complete a maneuver. The maneuver can be a climb, descent,turn, hover, take-off, landing, or any other maneuver input from aremote operator and/or as determined by a flight plan.

In some examples, the computing system 816 can include an imagemanagement system configured to calculate an anticipated vibrationalpattern based on the one or more control inputs required to complete amaneuver. In such examples, the computing system 816 can determine oneor more dead points of the anticipated vibrational pattern, and timewindows associated therewith. In some examples, the computing system 816can provide the one or more dead points of the anticipated vibrationalpattern and/or corresponding time windows to an image processing unit820 of the camera 806.

The inertial measurement unit 808 can measure an actual vibrationalpattern of the UAV 800. The inertial measurement unit 808 can determineone or more dead points of the actual vibrational pattern, and timewindows associated therewith. The inertial measurement unit 808 can sendthe one or more dead points and corresponding time windows to thecomputing system 814 and/or the image processing unit 820.

In various example, the image processing unit 820 can receive one ormore dead points (e.g., actual or anticipated vibrational pattern deadpoints) and/or corresponding time windows, and instruct the camera tocapture images by cycling a shutter during the one or more dead pointsand/or corresponding time windows. In such examples, the imageprocessing unit 820 can store the images captured by the camera. In someexamples, the image processing unit 820 can receive the one or more deadpoints and/or corresponding time windows, and can evaluate a timeassociated with one or more images previously captured by the camera.The image processing unit 820 can pre-process the one or more images bystoring the images corresponding to the one or more dead points and/orcorresponding time windows, and discarding the remaining images. Theimage processing unit 820 can send the stored images to the computingsystem 816.

In various examples, the computing system 816 can receive dead pointsand/or corresponding time windows from the inertial measurement unit808, and a plurality of captured images form the image processing unit820. In such examples, the computing system 816 can evaluate a timeassociated with each of the plurality of images captured by the camera.The computing system 816 can pre-process the one or more images bystoring the images corresponding to the one or more dead points and/orcorresponding time windows, and discarding the remaining images. In someexamples, the computing system 816 can store the images in a datastore818.

Additionally or alternatively, the functionality described herein can beperformed, at least in part, by one or more hardware logic componentssuch as accelerators. For example, and without limitation, illustrativetypes of hardware logic components that can be used includeField-programmable Gate Arrays (FPGAs), Application-specific IntegratedCircuits (ASICs), Application-specific Standard Products (ASSPs),System-on-a-chip systems (SOCs), Complex Programmable Logic Devices(CPLDs), etc. For example, an accelerator can represent a hybrid device,such as one from ZYLEX or ALTERA that includes a CPU course embedded inan FPGA fabric.

In various examples, the UAV 800 can include one or more communicationinterface(s) 810. In such examples, the one or more communicationinterface(s) 810 can enable wireless communication via wireless signalsbetween a remote computing device and the UAV 800. The wireless signalsmay include, but is not limited to, Bluetooth, radio control, voicecontrol, electromagnetic waves, Wi-Fi signals, cell phone signals, orsome combination thereof. In various examples, the one or morecommunication interface(s) 810 may facilitate the delivery of the one ormore processed images (e.g., clear images) to the remote computingdevice. In some examples, the one or more communication interface(s) 810may facilitate the receipt of one or more control inputs, such as from aremote operator.

In the context of software, the operations represent computer-executableinstructions stored on one or more computer-readable storage media that,when executed by one or more processors, enable the one or moreprocessors to perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,modules, components, data structures, and the like that performparticular functions or implement particular abstract data types. Theprocess can also be practiced in a distributed computing environmentwhere functions are performed by remote processing devices that arelinked through a communication network. In a distributed computingenvironment, computer-executable instructions can be located in localand/or remote computer storage media, including memory storage devices.

In the context of hardware, some or all of the blocks can representapplication specific integrated circuits (ASICs) or other physicalcomponents that perform the recited operations.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described. Rather,the specific features and acts are disclosed as illustrative forms ofimplementing the claims.

What is claimed is:
 1. A system comprising: one or more processors; andone or more memories coupled to the one or more processors, the one ormore memories storing instructions executable by the one or moreprocessors to perform acts comprising: measuring, by an inertialmeasurement unit of a vehicle, a vibrational pattern imposed on a cameraof the vehicle; determining, by a computing device of the vehicle, oneor more dead points associated with the vibrational pattern, wherein theone or more dead points comprise points of minimal acceleration in thevibrational pattern; identifying, by the computing device, a time windowcorresponding to a dead point of the one or more dead points; capturing,by the camera, an image based at least in part on the time window;storing the image in a cache of the camera; and sending the image to adatastore.
 2. The system of claim 1, wherein the capturing the imagecomprises causing a shutter of the camera to be open during the timewindow.
 3. The system of claim 1, the acts further comprising: openingand closing a shutter of the camera to enable the camera to capture aplurality of images, wherein the image is an image of the plurality ofimages; comparing times associated with each image of the plurality ofimages to the time window; and determining that a time associated withthe image corresponds to the time window, wherein the sending the imageto the datastore is based at least in part on the time associated withthe image corresponding to the time window.
 4. The system of claim 1,wherein the image is an underexposed image of an object, the actsfurther comprising: capturing a plurality of underexposed images of theobject; identifying one or more time windows corresponding to the one ormore dead points; determining that a first time associated with a firstunderexposed image of the plurality of underexposed images correspondsto a first time window of the one or more time windows; determining thata second time associated with a second underexposed image of theplurality of underexposed images corresponds to a second time window ofthe one or more time windows; and combining information from the firstunderexposed image and the second underexposed image to adjust anexposure of the image.
 5. The system of claim 1, the acts furthercomprising: identifying, by the computing device, a change in a path ofthe camera; predicting, by the computing device, an anticipatedvibrational pattern based at least in part on the change in the path;determining, by the computing device, one or more anticipated deadpoints of the anticipated vibrational pattern; and capturing a secondimage based at least in part on the one or more anticipated dead points.6. An unmanned aerial vehicle (UAV) comprising: an airframe; a cameracoupled to the airframe; an inertial measurement unit (IMU); one or moreprocessors; and one or more memories storing instructions executable bythe one or more processors to perform acts comprising: measuring, by theIMU, a vibrational pattern imposed on the camera; determining, by theIMU, one or more dead points associated with the vibrational pattern,wherein the one or more dead points comprise points of minimalacceleration in the vibrational pattern measured by the IMU;identifying, by the IMU, one or more time windows corresponding to theone or more dead points; and selecting an image based at least in parton a time of capture of the image corresponding to at least one of theone or more time windows.
 7. The UAV of claim 6, wherein the selectingthe image comprises: causing a shutter of the camera to be open during atime window of the one or more time windows; and storing the image in adatastore.
 8. The UAV of claim 6, the acts further comprising: sending,from the IMU to the camera, a signal comprising at least one of the oneor more dead points or the one or more time windows; and capturing theimage based at least in part on the signal.
 9. The UAV of claim 6,wherein the selecting the image comprises: capturing a plurality ofimages by the camera; identifying a first time associated with the imageof the plurality of images captured by the camera; and comparing thefirst time associated with the image to the one or more time windows.10. The UAV of claim 6, wherein the image is an underexposed image of anobject, the acts further comprising: capturing a plurality ofunderexposed images of the object; determining that a first timeassociated with a first underexposed image of the plurality ofunderexposed images corresponds to a first time window of the one ormore time windows; determining that a second time associated with asecond underexposed image of the plurality of underexposed imagescorresponds to a second time window of the one or more time windows; andcombining information from the first underexposed image and the secondunderexposed image to adjust an exposure of the image.
 11. The UAV ofclaim 6, the acts further comprising: identifying, by a computing deviceof the UAV, a change in a flight path; predicting, by the computingdevice, an anticipated vibrational pattern based at least in part on thechange in the flight path; determining, by the computing device, one ormore anticipated dead points of the anticipated vibrational pattern; andcapturing one or more images based at least in part on the one or moreanticipated dead points.
 12. The system of claim 11, wherein thepredicting the anticipated vibrational pattern further comprises:identifying, by the computing device, a maneuver associated with thechange in the flight path; determining, by the computing device, aflight control input used to effect the maneuver; and identifying, bythe computing device, one or more vibrations caused by the flightcontrol input.
 13. The UAV of claim 6, wherein the IMU is mounted in thecamera.
 14. The UAV of claim 6, wherein the IMU is mounted in theairframe proximate to the camera.
 15. A method comprising: identifying,by an inertial measurement unit on a vehicle, a vibrational pattern;determining, by a computing device on the vehicle, one or more deadpoints of the vibrational pattern, wherein the one or more dead pointscomprise points of minimal acceleration in the vibrational pattern;determining, by the computing device, one or more time windowsassociated with the one or more dead points; and storing an image in acache of a camera based at least in part on the one or more dead points.16. The method of claim 15, further comprising: opening and closing ashutter of the camera at a time corresponding to a time window of theone or more time windows, wherein a cycle of the shutter enables thecamera to capture the image, and sending the image to a datastore. 17.The method of claim 15, further comprising: capturing a plurality ofimages by the camera; identifying a time associated with each image ofthe plurality of images captured by the camera; and comparing the timeassociated with each image to the one or more time windows, wherein thestoring the image in the cache of the camera is based at least in parton the time associated with each image corresponding to a time window ofthe one or more time windows.
 18. The method of claim 15, furthercomprising: receiving a first signal from an image sensor; responsive toreceiving the first signal, opening an electronic shutter of the camerafor a first period and capturing light on a plurality of pixels;determining a sequence of the one or more time windows; and collecting agroup of the plurality of pixels corresponding to the sequence of theone or more time windows, wherein the group of the plurality of pixelscorresponds to the image stored in the cache of the camera.
 19. Themethod of claim 18, further comprising: determining that the image is anunderexposed image; and receiving a second signal from the image sensorbased at least in part on a determination that the image isunderexposed, wherein the second signal comprises an instruction to openthe shutter for a second period, the second period being longer than thefirst period.
 20. The method of claim 15, further comprising:identifying, by the computing device, a change in a path of the camera;predicting, by the computing device, an anticipated vibrational patternbased at least in part on the change in the path, wherein the predictingthe anticipated vibrational pattern comprises: identify a maneuverassociated with the change in the path; determining an input used toeffect the maneuver; and identifying one or more vibrations caused bythe input; determining, by the computing device, one or more anticipateddead points of the anticipated vibrational pattern; determining, by thecomputing device, an anticipated sequence of one or more anticipatedtime windows corresponding to the one or more anticipated dead points;and capturing, by the camera, images based at least in part on theanticipated sequence.